traffic calculator with hardcoded baseline #251
Conversation
…-UID2-6146-update-traffic-calculator-to-hardcoded-baseline
Ian-Nara
changed the base branch from
main
to
ian-UID2-6337-asynchronous-full-queue-process
…-UID2-6146-update-traffic-calculator-to-hardcoded-baseline
…-UID2-6146-update-traffic-calculator-to-hardcoded-baseline
…seline' into ian-UID2-6151-add-traffic-filter-class
Ian-Nara
changed the title
| /** | ||
| * Calculate total duration of allowlist ranges that overlap with the given time window. | ||
| */ | ||
| long getAllowlistDuration(long t, long windowStart) { |
There was a problem hiding this comment.
We should either validate that the allow list has no overlaps, or regenerate it as a new deduplicated allow list.
| LOGGER.info("Traffic calculation: oldestQueueTs={}", oldestQueueTs); | ||
|
|
||
| // Define start time of the delta evaluation window [newestDeltaTs - 24h, newestDeltaTs] | ||
| long deltaWindowStart = newestDeltaTs - this.evaluationWindowSeconds - getAllowlistDuration(newestDeltaTs, newestDeltaTs - this.evaluationWindowSeconds); |
There was a problem hiding this comment.
If evaluationWindowSeconds is 5 hours, the original range is [3, 8], and the allow list to exclude is {[2, 3], [6, 7]}, then the start time should be 1, not 2.
|
|
||
| // Count SQS messages in [oldestQueueTs, oldestQueueTs + 5m] | ||
| if (sqsMessages != null && !sqsMessages.isEmpty()) { | ||
| int sqsCount = countSqsMessages(sqsMessages, oldestQueueTs); |
There was a problem hiding this comment.
After the change, we’ll remove the fixed 5-minute window and instead process SQS messages until either 5 minutes have passed or 10k messages have been read, whichever comes first.
…-UID2-6146-update-traffic-calculator-to-hardcoded-baseline
…seline' into ian-UID2-6151-add-traffic-filter-class
* Temporarily Suppress libpng CVE-2025-64720 and CVE-2025-65018 * Upgrade uid2-shared
|
|
||
| } | ||
|
|
||
| LOGGER.info("SQS messages: {} in window [oldestQueueTs={}, oldestQueueTs+5m={}]", count, oldestQueueTs, windowEnd); |
There was a problem hiding this comment.
This needs to be updated to process all SQS messages in a single loop instead of only those from a 5-minute window. Just in case we get 1 billion requests, we stop after processing 10k messages rather than strictly using a 5-minute window.
There was a problem hiding this comment.
It is handled by SqsWindowReader
https://github.com/IABTechLab/uid2-optout/pull/254/files
There was a problem hiding this comment.
Got it. So if processing 10k messages takes about 3 minutes, we’ll still scan the full 5-minute window, but after 3 minutes there simply won’t be any more messages to read, is it correct?
…-UID2-6146-update-traffic-calculator-to-hardcoded-baseline
…seline' into ian-UID2-6151-add-traffic-filter-class
…ter-class add traffic filter class
Ian-Nara
merged commit 4388f1b
into
ian-UID2-6337-asynchronous-full-queue-process
Ian-Nara
deleted the
ian-UID2-6146-update-traffic-calculator-to-hardcoded-baseline
branch
No description provided.